SPyChart - Example notebook

---

# Importing SPC class
from SPyChart import SPC
# Pandas and numpy to create example input time series datasets
import pandas as pd
import numpy as np
from plotly import graph_objects as go
import plotly.express as px
from plotly.subplots import make_subplots

Note: This tool requires a dataset with a minimum of 15 data points in order to function. Additionally, an additional column 'n' will need to be included (representing sample size) if using the following charts: - p-chart - np-chart (n must be fixed) - u-chart

"""
Artificial daily data with three distinct process changes, sampled from a Normal distribution.
"""

# Empty dataframes
df_1 = pd.DataFrame()
df_2 = pd.DataFrame()
df_3 = pd.DataFrame()

# Create each dataframe sampling from normal dist. with different params.
df_1['ds'] = pd.date_range(start='2020-01-01', end='2020-03-01', freq='d')
df_1['TARGET'] = np.random.normal(loc=50, scale=10, size=len(df_1))
df_2['ds'] = pd.date_range(start='2020-03-02', end='2020-06-01', freq='d')
df_2['TARGET'] = np.random.normal(loc=40, scale=6, size=len(df_2))
df_3['ds'] = pd.date_range(start='2020-06-02', end='2020-09-01', freq='d')
df_3['TARGET'] = np.random.normal(loc=60, scale=7, size=len(df_3))

# Combine all datasets into one, with clear process changes at 2020-03-02 and 2020-06-02.
df = pd.concat([df_1, df_2, df_3])

df = df.set_index('ds', drop=True)

# Show first five rows.
df.head()

	TARGET
ds
2020-01-01	47.099271
2020-01-02	50.105789
2020-01-03	43.319262
2020-01-04	43.987127
2020-01-05	53.548542

Basic SPC Analysis

In this example, we will create a simple SPC chart. For this example, we will ignore any process changes and plot the SPC chart with only a single calculation for the control lines (i.e., the control lines will take all data into account).

$XmR - chart$

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='XmR-chart') # We will use an XmR chart in this example
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (XmR-chart) - Basic usage')

# If we want the data to build our own SPC charts, use the spc_data attribute.

# Note this returns 2 dataframes.
#    - Data for the individual Chart (top chart)
#    - Data for the mR Chart (bottom chart)

spc_data_X, spc_data_Y = spc_example.spc_data
spc_data_X.tail()

	ds	TARGET	cl	lcl	ucl	+1sd	-1sd	+2sd	-2sd	Rule 1 violation	Rule 2 violation	Rule 3 violation	Rule 4 violation	Rule 5 violation	chart type
240	2020-08-28	53.615789	50.579165	28.762026	72.396304	57.851545	43.306785	65.123924	36.034406	0	1	0	0	0	X-chart
241	2020-08-29	65.293999	50.579165	28.762026	72.396304	57.851545	43.306785	65.123924	36.034406	0	1	0	0	0	X-chart
242	2020-08-30	59.063933	50.579165	28.762026	72.396304	57.851545	43.306785	65.123924	36.034406	0	1	0	0	0	X-chart
243	2020-08-31	71.219245	50.579165	28.762026	72.396304	57.851545	43.306785	65.123924	36.034406	0	1	0	1	0	X-chart
244	2020-09-01	54.656169	50.579165	28.762026	72.396304	57.851545	43.306785	65.123924	36.034406	0	1	0	0	0	X-chart

# Note that spc_data_Y will be None in SPC charts that only include a single chart (i.e. Individual-chart).
spc_data_Y.head()

	ds	TARGET	r	cl	lcl	ucl	+1sd	-1sd	+2sd	-2sd	Rule 1 violation	Rule 2 violation	Rule 3 violation	chart type
0	2020-01-01	47.099271	NaN	8.203244	0	35.027852	-18.621364	35.027852	-45.445973	61.852461	0	0	0	mR-chart
1	2020-01-02	50.105789	3.006518	8.203244	0	35.027852	-18.621364	35.027852	-45.445973	61.852461	0	0	0	mR-chart
2	2020-01-03	43.319262	6.786527	8.203244	0	35.027852	-18.621364	35.027852	-45.445973	61.852461	0	0	0	mR-chart
3	2020-01-04	43.987127	0.667865	8.203244	0	35.027852	-18.621364	35.027852	-45.445973	61.852461	0	0	0	mR-chart
4	2020-01-05	53.548542	9.561415	8.203244	0	35.027852	-18.621364	35.027852	-45.445973	61.852461	0	0	0	mR-chart

Re-calculating control lines after each process change.

If you know a process change is occuring, and want to re-calculate the control lines to reflect the change, we can include the change_dates argument (must be a list of date(s)) in the initialisation, shown below.

$XmR -chart$

spc_example = SPC(data_in=df, # data we created above
                  target_col='TARGET', # name of column to analyse
                  chart_type='XmR-chart',
                  change_dates=['2020-03-02', '2020-06-02']) # We will specify change date (needs to be in a list)
# change_dates can include as many dates as necessary, but must be in a list.
spc_example.setup()
spc_example.check_rules()
spc_example.plot_spc(title='Example SPC (XmR-chart) - Re-calculating control lines')

Calculating control lines on baseline data

Now, suppose we didn't know a whether a change in the system had an impact on our measured process. Rather than using the change_dates parameter, we can use the baseline_date parameter, to calculate control lines only on data before this specified date. This will give an indication of the impact to the measured process following a change to the system.

$Individual-chart$

spc_example = SPC(data_in=df, # data we created above
                  target_col='TARGET', # name of column to analyse
                  chart_type='Individual-chart', 
                  baseline_date='2020-03-01') # we will specify baseline date
spc_example.setup()
spc_example.check_rules()
spc_example.plot_spc(title='Example SPC (Individual-chart) - Setting baseline date')

Non-daily data

Our examples have used daily data, but data of any frequency can be used.

"""
Artificial hourly data with three distinct process changes, sampled from a Poisson distribution.
"""

df = pd.DataFrame()
df['ds'] = pd.date_range(start='2020-01-01', end='2020-02-01', freq='H')
df['TARGET'] = np.random.poisson(lam = 10, size=len(df))
df = df.set_index('ds', drop=True)
# Print first few rows
df.head()

	TARGET
ds
2020-01-01 00:00:00	14
2020-01-01 01:00:00	10
2020-01-01 02:00:00	7
2020-01-01 03:00:00	10
2020-01-01 04:00:00	10

$c-chart$

spc_example = SPC(data_in=df, 
                  target_col='TARGET', 
                  chart_type='c-chart') 
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (c-chart) - Hourly data')

Other SPC types

Some SPC charts take averages of samples of data, meaning each day (or hour, week, etc...), has n samples.

# Creating dataset
df_1 = pd.DataFrame()
df_2 = pd.DataFrame()
df_1['ds'] = list(pd.date_range(start='2020-01-01', end='2020-03-01', freq='w'))*2
df_1['TARGET'] = np.random.normal(loc=40, scale=10, size=len(df_1))
df_2['ds'] = list(pd.date_range(start='2020-03-02', end='2020-09-01', freq='w'))*2
df_2['TARGET'] = np.random.normal(loc=65, scale=10, size=len(df_2))
df = pd.concat([df_1, df_2], axis = 0).sort_values(by = 'ds').reset_index(drop=True)

df = df.set_index('ds', drop=True)

df.head()

	TARGET
ds
2020-01-05	51.031174
2020-01-05	21.659477
2020-01-12	57.371241
2020-01-12	38.151974
2020-01-19	38.509114

$\bar{X}R - chart$

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='XbarR-chart', 
                  change_dates=['2020-03-01'])
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (X bar R -chart) - Re-calculating control lines')

Duplicate dates detected.
Constant sample size = 2

Generally a sample size $>=5$ qualifies the use of the $\bar{X}S - chart$ (standard deviation rather than range)

$\bar{X}S - chart$

# Weekly data
df = pd.DataFrame()
df['ds'] = list(pd.date_range(start='2020-01-01', end='2020-06-01', freq='W'))*10
df['TARGET'] = np.random.normal(loc=50, scale=10, size=len(df))
df = df.set_index('ds', drop=True)

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='XbarS-chart')
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (X bar S -chart) - Basic usage')

Duplicate dates detected.
Constant sample size = 10

With a p-charts, u-charts and np-charts, you'll need to feed in an additional 'n' column, represnting the sample size, in order to calulate proportions. Since we need to know n at each position, the baseline approach would no longer be suitable.

$p - chart$

df = pd.DataFrame()
df['ds'] = list(pd.date_range(start='2020-01-01', end='2020-01-15', freq='d'))
df['TARGET'] = np.random.randint(5, 20, len(df))
df['n'] = np.random.randint(50, 80, len(df))
df = df.set_index('ds', drop=True)

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='p-chart') 
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (p-chart) - Basic usage')

$np - chart$

df = pd.DataFrame()
df['ds'] = list(pd.date_range(start='2020-01-01', end='2020-01-15', freq='d'))
df['TARGET'] = np.random.randint(1, 10, len(df))
df['n'] = 50
df = df.set_index('ds', drop=True)

spc_example = SPC(data_in=df, 
                  target_col='TARGET', 
                  chart_type='np-chart') 
spc_example.setup()
spc_example.check_rules()
spc_example.plot_spc(title='Example SPC (np-chart) - Basic usage')

$u - chart$

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='u-chart')
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (u-chart) - Basic usage')

# Creating another dataset.
df_1 = pd.DataFrame()
df_2 = pd.DataFrame()
df_1['ds'] = list(pd.date_range(start='2020-01-01', end='2020-01-15', freq='d'))
df_2['ds'] = list(pd.date_range(start='2020-01-16', end='2020-01-30', freq='d'))
df_1['TARGET'] = np.random.randint(5, 20, len(df_1))
df_2['TARGET'] = np.random.randint(25, 40, len(df_2))
df_1['n'] = np.random.randint(50, 80, len(df_1))
df_2['n'] = np.random.randint(40, 50, len(df_2))
df = pd.concat([df_1, df_2], axis = 0).reset_index(drop=True)
df = df.set_index('ds', drop=True)
df.head()

	TARGET	n
ds
2020-01-01	11	54
2020-01-02	19	62
2020-01-03	5	64
2020-01-04	5	79
2020-01-05	7	79

$u - chart$ (re-calculating control lines)

spc_example = SPC(data_in=df, # data we created above (input dataset)
                  target_col='TARGET', # name of column to analyse (existing within df)
                  chart_type='u-chart', 
                  change_dates=['2020-01-15'])
# First method to calculate control lines and store the data within the class.
spc_example.setup()
# Next, we call the check_rules() method to test SPC rules, using the above calculated control lines.
spc_example.check_rules()
# Plot the resulting chart in Plotly (interactive & downloadable plot)
spc_example.plot_spc(title='Example SPC (u-chart) - Recalculating control limits after process change.')